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Locally-Directed 
Evaluation Resources 



The resources available for aiding-in the planning and con^ 
duct of Locally Direcfed Evaluations include: (1) Local 
Leader Guides, and (2) Consultant Assistance. ^ 

Local Leader Guides 

There are 12 different logal leader guides. The set in- 
cludes the following Local Leader Guides: 

Student Evaluation of Instruction, 
Student/Employer Follow-up 
Measuring Student Competencies - 
Assessing Basic Skills 
Assessing Student Services 
Assessing Special Needs Services and 
Identification Practices 
Assessing Access and Eqtfity 
Evaluating Facilities 
Evaluating Instructional Materials ' 
CostOutconne Analysis 
Analyzing Community ^Resources 
Assessing Community Needs 

The term "local leader" is used to identify the person who 
leads or directs a particular activity. LocaMeaders can be 
administrators, guidance personnel, instructors, advisory 
committee memt^ers, and other personnel. 

Each Local Leader Guide is designed to help the leader 
coordinate the activity. Local Leader Guides have three (3) 
major parts. Each part is briefly described below: 
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Supporting Documents. This sectiOQ contains example 
forms, infomr^tion handouts, copimuniques, and in- 
struments that c^ be utilized in their present form or* 
might be helpful in the design of new ones to meet in- 
dividual institutional needs. These documents have been 
' developed and used in educational institutions in conjunc- 
tion with the testing of the evaluation tasks. References 
for additional supporting matenal are also provided with 
sane of the activities 

Consultant Assistance 

A team of consultants within the Illinois State Board of 
Education, Department of Adult, Vocational and Technical 
Education have been trained' as ^|luation consultants 
and are available to help initiate il^ facilitate Evaluation 
efforts within local educational agencies (community col- 
leges, area vocational centers, high schools, elementary 

, schools and special agencies) at no cost to LEA's. These 
consultants can help, you in developing an evaluation 
system and in conducting individual activities within the 
system. They can provide assistance in three ways 1) by 

' visiting your LEA 2) by talking with you by telephone, and 
3),by communicating with you through the mail. Most like- 
ly a combination of the approaches will be used. 

Consultant assistance can be gained by making a request 
to the Manager of the Consultant Sen/ices Section of the 
Department of Adult. Vocational and Technical Education. 
Illinois State Board of Education. ' 



Preliminary Considerations. This section offers sugges- 
tions regarding the initiation or planning o( an activity. It 
generally offers suggestions for holding a rpeeting to in- 
form staff and potential team memt^ers of the purpose and 
procedures of the activity. 

Procedures/Task Breakdown. This^s^otion contains a 
specific listing of suggested t&sks and sub-tasks that pro- 
vide a structure for conducting the activity. This section is 
the substancej^f the activity. The suggested procedures 
have been tried out in many educational institutions in 
field test settings. 



Description of Locally- 
Directed Evaluation 
Activities 



The following pages provide an overview of each of the 
twelve Locally-Directed Evaluation activities. These entries 
have been provided to help familiarize you with each ac- 
tivity. An activity description, a list of suggested purposes, 
suggested methods and suggested events are included 

Activity #1 Student Evaluation of Instmctlon 
Activity Description: . m 

This activity is designed to aid LEA staff in obtaining infor- 
mation froni current students to facilitate the improvement 
of classroom instruction This activity suggests that a team 
of staff develops a questionnaire or rating form to be ad- 
ministered to currently enrolled students The question- 
naire results are then summarized and each instructor 
(with support from others) prepares a report which in- 
cludes suggestfons for improving the instructional process 
This activity can be conducted for the total LEA. occupa- 
tional program, individual programs, or individual courses 
Emphasis is placed on individual instructor use of results 



Suggested Purposes: 

1. To encourage teacher initiated improvements to the 
process of instruction and learning 

2. To improve courses through more effective lesson plan- 
ning. 

3. To provide information that will improve provisions for 
individual differences. 

4 To improve offerings for special needs students 

5. To aid students in choosing courses and instructors 

6. To indicate areas of in-service needs to improve 
teacher competency. 



Suggested Mefhods: 

Student questionnaires and rating scales 

Suggested Events: 

Hold staff meeting 
Define scope 
Construct instruments 
Administer instruments 
Collate and tally data 

Write conclusions, recommendations and suggested 
solutions 

Discuss, adopt, adapt, implement recommendations and 

suggested solutions 
Report findings 



Activity #2 Student/femployer Follow-Up 
.Activity Description: 

This activity accomplishes the dual purposes of learning 
about instructional programs and services from former 
students and their employers. Tj^e.student follc5w-up can 
help to determine the effectiveness of programs in terms 
of job placement, a product evaluation measure. The 
survey should be based upon the need for specific pro- 
gram improvement information that will aid in answering 
broad key questions formulated by staff 

The^empldyer survey involves the contact of employers to 
obtain their.rating and comments on the performance of 
former students The information gathered should answer 
prespecified key questions formulated by staff A local 
leader and a team of staff and advisory commitjee 
members should plan and coordinate this follow-up activi- 
ty This activity. can focgs on the total LEA, vocational pro- 
gram area or individual programs or courses 



Suggested Purposes: 

1 To determine the adequacy of the educational or train- 
ing program in preparing individuals for job entry 

2 To determine the adequacy of preparation for entry into 
advanced training such as industrial, training programs, 
higher education, or continuing education programs. 

3 To determine the mobility of program graduates. 

4 "Yo obtain suggestions for program improvement from 
former students, 

5 To assess the current performance of former students. 
6. To elicit employer recommendations for improving pro- 

• grams 

7 To aid in public relations. 

Suggested Methods: 

Personal Interview 
'Telephone Interview 
Mail Questionnaire 



Suggested Events: 

Hold staff meeting 

Choose team members 

Make a schedule and assign tasks 

Choose or develop and test instruments 

Administer instruments 

Summarize and present results 

Obtain recommendations and suggestions 

Utilize results 



Activity #3 Measuring Student Competencies 
Activity Description: 

Jh^activtty is designed to help instructors improve and 
expand upon their techniques for assessing student attain 
ment of competencies Also, the increased use of 
measurement results by instructors, administrators, 
employers and students is stressed. The actmty involves 
the development of instruments and measurement techni- 
ques to assess various forms of objectives within the 
cognitive, affective and psychomotor domains It also in- 
volves the preRaration of individual and group profile forms 
that can -be helpful in using the results. Instructors should 
assume the pnmary responsibility for completing this ac- 
tivity and cooperation among instructors who teach the 
same courses or subjects is encouraged The activity can 
be done by all instructors or individual instructors can use 
It independently * 



Suggested Purposes: 

1 To provide a more comprehensive approach to the 
evaluation of student learning 

2 To aid in placing learners in programs commensurate 
with their interests and abilities 

3 tTo assist in assessing the instruction, course or pro* 

gram competencies. 

4 Tg^^upgrade staff competencies in the principles and 
techniques of measuring student achievement. 

5 To idfentify problem areas of instruction 

6. To identify gifted students or special needs students. 



essential to the successful completion of vocational pro- 
grams are identified. Criterion or non»referenced tests 
which assess students' basic skills are selected or con- 
structed and administered to students. LEA vocational 
staff can then identify and work with students who need 
improvement in Jj^sic skills areas. This activity can be ap- 
plied to the total LEA, the vocational program, or individual 
programs and courses. 



Suggested Purposes: 

1 To identify basic skills in the areas of reading, writing, 
anthmetic and oral communications essential in voca- 
tional programs. 

2. To analyze course materials to inventory basic skills 
learning expenences in vocational courses. 

3 To select or develop and administer basic-skills tests to 
students to identify areas of weakness. 

4. To inform internal and external groups of the role of 
vocational education in basic skills education. 

5 Yo provide instruction in basic skills to students 1o help 
increase their chances of success in vocational pro- 
grams. 

6 To conduct in«service activities to help LEA staff utilize 
basic skills test results to help students. 



Suggested Methods: 

Construct Basic SkillsA/ocational Course Grids 
Select or develop Basic Skills tests 
Administer tests 



Suggested Methods': 

Development of measurerinent items 
Developnfient of student profiles 



Suggested Events: 

Hold staff meeting 
Develop adequate objectives 
Develop instruments 

Develop student performance record forms 
Develop a system of reporting 



Suggested Events: 

Hold staff meeting 

Define the assessment scope 

Decide who should conduct the course assessment 

Construct grids 

Make status reports 

Form a Testing Task Force 

Select or construct Basic Skills tests 

Administer tests 

Prepare a written report of results 

Make recommendations and suggested improvements 

In-service staff who will use test results 

Utilize results 



Activity #4 Assessing Basic Skills 
Activity Description: 

This activity is designed to assess vocational students' 
basic skills in the areas of reading, writing, mathematics 
and oral communications. Results of this activity aid in 
making professional decisions regarding remediation, in 
dividualized learning, program and course improvements 
and for making status reports to various internal and ex 
ternal publics concerned yvith the contribution of voca 
tional education to improving basic skills. Basic skills 



Activity #5 Assessing Student Services 

Activity Description: 

This activity focuses on the evaluation of a broad array of 
services offered to students. The suggested procedures 
are applicable to the evaluation of student career interest 
services, the career information program and othef "com- 
mon student services. In the student career interest area, 
the major thrust is on local assessment of student in- 
terests using locally developed interest instruments. Also 
the quality, availability and use of career information ' 
materials is assessed. A combination of methods including 
questionnaires, interviews and observations are used to 
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gather perceptions and reactions from cyrrent students, 
former students, LEA staff, and employers. This activity 
can focus on all student services or on specific ones such 
as: counseling, career interest services, career information 
services, youth organizations or.placement services. 



Suggested Purposes: 

1 To determine student opinion regarding the availability 
and effectiveness of current services 

2. Tp determine the relevance of services provided and 
the efficiency of personnel utilization 

3. To determine community acceptance and support for 
services currently provided. 

4. To improve individual student career planning. 

5. To encourage the development of student self- i^ 
knowledge regarding attitudes, interests, abilities and 
values. 

6. To assess the organization and impact of career infor- 
mation. 



Suggested Purposes: 

1 To identify the needs of current and potential special 
learners. 

2 To irtventory services available to special needs 
learners. 

3 To compare needs v^ith available services. 

4. To obtain ratings of services from current and former 
students. 

5 To obtain ratings of services from staff and parents. 

6 To identify additional services that are needed. 

Suggested Methods: 

Interviews 

Questionnaires 

Ratings 

Suggested Events: 



Suggested Methods: 

Observation 
Personal interview 
Telephone interview 
Mail questionnaire 



Suggested Events: 

Hold staff meeting ' 

Select a team 

Develop criterion questions 

Select a method for obtaining information 

Develop forms or instruments ^ 

Administer instruments and collect data 

Tabulate the collected information 

Prepare a final report , 

Utilize results 



Activity #6 Assessing Special Needs 
Services and identification Practices 

Activity Description: 

This activity is designed to improve services to special 
needs students It is divided into two parts or phases. 
Phase one involves the identification and specification of 
the special needs of current and potential students and a 
comparison of needs to available services. The second 
phase involves determining the adequacy of services 
through the gathering of information from current and 
former students, staff, parents and community represen- 
tatives Interviews, questionnaires and observations are 
used to gather information about services. A team of in- 
dividuals should design and coordinate this activity, but 
staff awareness is necessary to ensure ease in data col- 
lection and care in the utilization of results. The activity 
should initially focus on the total LEA with subsequent use 
in individual programs. 



Hold staff meeting 
Establish^a team 

Orient the team and discuss issues 
Assess or reassess the needs of identified special 
students 

Inventory currently available or offered services 
Compare student needs with identified services. 
Obtain ratings of effectiveness of services from LEA staff 
Obtain ratings of effectiveness of services from current 

and former students 
Tabulate results 

Analyze data and prepare a report 

Utilize results to impr6ve programs and services 

Activity #7 Assessing Access and Equity 

Activity Description: 

This activity is designed to.assist LEA personnel assess 
vocational services. to special populations for the purpose 
of improving services and to evaluate programs and ser- 
vices to ensure compliance with the civil rights guidelines 
in vocational education The activity describes tasks in 
terms applicable to alt protected populations. Attitudinal * 
barriers, policy and practice limitations, communications 
barriers and facility use and equipment and transportation 
restrictions will be identified, if they exist This activity can 
be applied to the tgtal LEA, the total vocational program or 
a specific program 

Suggested Purposes: 

1 To insure that vocational programs and services are 
accessible to all students regardless of race, color, na- 
tional origin, sex or handicaps 

2 To identify and eliminate barriers, linjitations or restric- 
trons which might inhibit student enrollment in voca- 
tional programs. 

3 To better serve special populations. 

4. To gain familiarity with laws and rules and regulations 
impacting special needs students in vocational pro- 
grams. 



5. To implemertfimprovement^ in the services available to 
special needs students. 

6. To ensure compliance with the civil rights guidelines in 
vocational education. 

&iggested Methods: 

Analysis of laws, rules and regulations 
Interviews 

Questionnaires 
Checklists 



Suggested Events: 

Hold staff meeting 
Establish a team 
Review written documents ' 
Formulate a focus and purpose 
Identify key questions 
Develop evaluation instruments 
Collect data 
Analyze data 
Prepare a report . 

Prepare recommendations and suggested improvements 
Utilize results 



Activity #8 Evaluating Facilities 
Activity Description: 

This activity is designed to help in analyzing existing 
facilities for the purpose of determiningjuture needs for 
expansion, renovation or abandonment. Additionally, this 
activity can help in improving the use and safety of 
facilities. This activity involves the inventory of facilities 
and the observation" and rating of facility, characteristics by 
both advisory committee members and staff members. A 
local leader and a team of internal staff should plan and 
coordinate the evaluation and should prepare a report with 
reconnmendations This activity can be applied to the en- 
tire LEA, the occupational program or individual programs 
and courses. 

Suggested Purposes: 

1. To determine the adequacy and use of the existirig 
school plant 

2. To evaluate the LEA's site, buildings, and non- 
instructional equipment. 

3 To provide a basis for planning future utilization of 
school facilities. 

4. To plan for future renovation and modernization 

5. To determine priorities for* the correction of substandard 
' conditions. 

6. To promote, awareness of the. Federal Occupational 
Safety and Health Act (OSHA). 



Suggested Methods: 

Team observation 



Suggested Events: 

Hold staff meeting 

Establish a team to plan and coordinate 

Fomnalize purpose and scope 

Formulate key questions 

Choose appropriate type of study \ 

Conduct study selected 

Summarize information 

Prepare a report 

Activity #9 Evaluating Instructional Materials 

Activity Description: ^ 

This activity focuses on the evaluation of the LEA's collec- 
tion of instructional materials. The activity includes three 
phases that are designed to improve the collection and 
use of materials. These phases are: 1) the inventory of all 
instructional materials that are owned by the LEA, 2) an 
assessment of the adequacy of each of the materials; and 
3) the utilization of results of phases one and two to In- 
crease the use of materials and also aid in the selection 
of new nnatenals. A combination of methodB, including the 
survey, staff rating of materials, and studefit rating of 
materials, is suggested. A local leader (possibly a media 
director or librarian) directs a team of instructional and - 
support personnel. The activity is designed for assessing 
the LEA's entire collection of instructional materials 
although it can be adapted to focus on certain segments 
of the media collection. 



Suggested Purposes: 
/ • 

1 To Increase staff awareness o) currently owned instruc- 
tional materials. - 

2 To Increase the use of instructional materials. 

3 To identify and discard Ineffective instructional 
materials. 

4 To aid in identifying the need for additional materials. 

5 To evaluate new materials prior to purchase. 

6. To Improve instruction through the morjS effective use 
of instructional materials. 



Suggested Methods: 

Staff survey • 

Staff rating of materials 

Student rating of materials 



Suggested Events: 

Hold staff meeting 

Select coordinating team^ 

Conduct inventory of all institutional instructional 

materials 
Establish a central catalog file 
Select, develop or adapt* staff review^ Instruments 
Administer staff review instruments ^ 
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Select, develop or acJopt a student reaction form 

Administer student reaction fomas 

Summarize collected information 

Recofd and report results 

Utilize results— catalogjile 

Utilize results to improve decision making 

Activity #10 C6st/Outcome Analysis 
Activity Description: 

This activity is designed to aid LEA personnel in collecting 
information regarding costs and relating it to informatioa 
about outcomes. The activity can focus on the cost/out- 
come relationship of 1) a program, 2) a course; 3) a unit of 
instruction; and 4) a type of media. The results of the ac- 
tivity can assist decision making atxxjt instructional ^Iter- - 
natives and can help increase the efficiency of instruction. 
One part of the activity involves the collection of cost in- 
formation from instructional staff members and existing 
records. Outcome information is gained from existing 
measures, e.g., test scores, follow-up results, other ongo- 
ing Locally-Directed Evaluation activities and special data 
collection procedures. This activity can be applied to the 
total LEA, the occupationalprogram, individual programs, 
individual courses or more specific components. 



Suggested Purposes: 

1. To provide assistance in choosing among alternatives 
that relate to choices in programs, instructional techni- 
ques, and equipment and material purchases. 

2. To help justify expenditures to the administration, the 
board of control, the public, state and federal funding 
agents and to stildents. 

3. To justify resources allocation decisions. 

4. To decrease expenditures of high cost/low incidence 
programs. 

5. To determine optimum scheduling/sequencing. 

6. To determine the advisability of financing the develop- ' 
ment of a new program. 



Suggested Methods: 

Analysis of existing records 
Collection of outcome information 



Suggested Events: 

Hold staff meeting 

Establish a team to plan and coordinate 
Determine scope 

Develop or select cost accounting instrument ^ 

Collect cost data 4 

Summarize cost data 

Identify existing outcome information 

Collect and summarize outcome information 

Relate cost information to outcome information 

Prepare a written report 

Utilize results 



Activity #11 Analyzing Community 
Resources 

Activity Description: 

^ This activity is designed to aid in evaluation of the 
availability and effectiveness of resources such as ad- 
visory committees, speakers, fi^, trip?, etc. The activity 
has two parts. Part one focuses on the identification of 
available resources through the use of a mail question- 
naire or a telephone survey. The result of part one is the 
establishment of a resource file or catalog. Part two 
focuses on the rating of resources by students and staff 
through the use of a printed questipnnaire or rating scale. 
The results of part two can be irfQorporated into the 
resource file and utilized to improve the selection of 
outstanding resources A team-of individuals consisting'of 
administrative staff, instructors, and advisory committee 
members coordinate the activity If possible, this activity 
should be applied to the total LEA. however, it can be ap- 
plied to any instructional component of the LEA. 

Suggested Purposes: 

1 To develop a comprehensive listing of available\:om- 
munity resources anS lo establish a convenient central 
file system. 

2. To establish an ongoing evaluation system for 
resources. 

3 To improve continuing educatidrl programs. 

4 To promote communications t)etween all segments of 
the community. 

5 To revise progranrls based on Changing needs withirr 
the community. 

6. To improve placement for program leavers. 

Suggested Methods: * 

Personal interviews 
Telephone interviews - *^ ^' 
Mail questionnaire 

Teacher ratings ^ . ' * * 

Student ratings ,^ > 



Suggested Events: ^ ^ " ^ 

Hold staff meeting . ^ 

Develop an instrument to determine current resource 
utilization 

Develop a qualitative rating instrument of resources cur- ^ 
rently utilized 

Develop a survey instrument to identify additional available 

resources . 
Preparing for and administering data collection 
Summarizing and presenting the results 
Coordinate use of collected data and updating 



Activity #12 Assessing Community Needs 



Activity Description: 



This activity suggests a method to identify the vocational 
progranns needed in a local community. It provides the 
LEA with a way of monitoring student interests, communi- 
ty needs and citizen expectations. The need for existing or 
n^ programs is assessed in a systematic manner. A 
systennatic survey approach to assessing local community 
needs provides information and involvement from diverse 
community groups in the LEA*s planning effort. The^sug- 
gested needs assessment system involves person^ inter- 
views, telephone inten/iews afkJ mail surveys to determine 
community needs. This activity can be applied to the en- ^ 
tire community or certain community segments. 



1^ To identify community needs tind interests with respect 
'to local vocational education programs offered by Ihe 



2. To project future vocational programs which will rtieet 
the employment heeds of local employers. 

3. To gain systematic, continuous feedback and com- 
munications ^rom local community members or groups. 

4. To identify potential student interests in present and 
future. programs. 

5. To identify any discrepancies between the needs^ of the 
community and the programs available to meet those 

^ needs. 

6. To help establish, better relations with the business/in- - 
dustrjal community andcjtizens groups. 

Suggested Method^: ^ / 

Person^.1 interviews 
Telephone interviews . 

Mail surveys ^ . " ^ 



Hold staff, pneeting 
' Establish a coordinating t^am 

Determine scope and goals of 'the assessment 

Develop key question^. 

Identify specific target groups 

Construct or adapt survey instrument 
' Conduct pre-test 

Cbllect data 

Analyze data 

Prepare written report 

Communicate report' ^ 

Implement results • 




LEA. 



Suggested Events: 




Locally Directed 
Evaluation— Getting 
Ready 



Each local education agency (LEA) will have other unique 
"reasons why" they should conduct evaluations. Local 
needs will supplement the five (5) "reasons why" 
previously lisfed. * ^ 

An innportant second step in the evaluation journey is to 
deternnine "what to evaluate." 



All journeys, projects and evaluations begin with a first 
step. The first stepjn conducting Lbfcally Directed Evalua- 
tions is to answer the question, "Why evaluate?" 
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Locally DIrecttd Evaluation 
Reasons to Evaluate 



Why Evaluate? 

Some of the more obvious reasons are as follows: 

1. TO/ improve Programs 

Evaluation of programs will nearly always contribute to 
serving students better than in the past. If evaluation of- 
fers bo\}f] value judgements regarding what is happen- 
ing within programs and constructive suggestions for 
improvement, then the services and programs offered 
to students will improve. This "reason why" is the most 
important goal of evaluation efforts. The "bottom line" 
of all evaluation activities \^ \q improve 

2. To Aid in Decision Making 

Decisions must be based on a certain amount of ac- 
curate information. Those important decisions regarding 
program changes, additions or delations, selection of 
students and the selection of alternatives all require 
that objective information be gathered and analyzed so 
the "right" decision can be madS. Decisions based on . 
intuition only often lead to undesirable outcomes. Ra- 
tional decisions based on evaluative information are 
defensible and justifiable. 

3. To improve Planning 

Evaluation can provide information which measures 
resources, limitations and possibilities as well as 
assessing current status. This information can be of 
use in developing plans. Local education agencies need 
to formally assess where they are before they decide 
* where they want to go. 

4. To insure Accountability 

Accountability to significant internal and external* forces 
requires a presentation of a program's results in rela- 
tion to established objectives and incurred costs. The 
results of successful program, compfeters, placement of 
students in jobs, effective administration and services 
are important to someone or group. A well planned and 
implemented evaluation activity can help meet accoun- 
tability requirements. 

5. Tcfvlmprove Personnel > 

Evaluation results, and the activity of evaluation, are 
'strong staff development focuses. Staff members learn 
from evaluations. By taking an active role in evaluations 
and by analyzing the results of evaluations, staff can 
change for the better. Evaluation, done correctly, is a 
strong motivating force. The motivation emanates from 
within the staff membersr themselves and not from 
^ some external force. 



Improve 
programs 




What Shotild be Evaluated? 

The answers to this question will depend, to a large ex- 
tent, on local needs. There are, however, four (4) broad 
forms of evaJuations which can help an LEA partially - 
answer this second question. These forms are often refer- 
red to as the CIPP model. 

' C = Context 
I = Input 
P = Process 
P = Product 

The objectives of these four (4) forms of evaluation are 
listed as follows: 



Evaluation Fonm 

1. Context Evaluation 



2. Input Evaluation 
3 Process Evaluation 

4. Product Evaluation 
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Objective 

To define the educational environ- 
ment, to identify student needs, to 
delineate problems and to formulate 
goals. 

To identify and assess alternative 
' strategies for achieving goals. 

To observe program operation 
ascertaining procedural events and 
activities to identify problems In 
design and implementation. 

To a'ssess program outcomes and 
to relate these outcomes to objec- 
tives. 
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The specific evaluation activities asscx:iated with Locally 
Directed Evaluation address all of these evaluation forms. 
Again, the local situation will detemrime which objectives 
should be addressed by the specific activities. 

The third question to be dealt with is "who should be in- 
volved in conducting evaluatidns?" 

Tlgure 2 

Locally Directed '^valuation 

What to Evaluate * ' 

; ' Product 



Process 



tnput 



A broad-based evaluation effort accomplishes the follow- 
ing desirable goals: 

a. Spreads work loads among a large number of people 
not overloading any small grou?^ 

b. Provides for group input thus conskJering many points 
of view and prior experiences and knowledge. 

^c. dh/es a sense of ownership to a wide variety of people 

representing a number of different groups, 
d. Lends authority, by way of numbers and types of peo- 
- pie involved, to the resujts of the evaluation undertak- 
ing. 

A rule of thumb to use in asisembKng an evaluation team 
is to involve those individuals who have an interest in the 
program or service which is to be evaluated. Their per- 
sonaf interests will often complement the interests of the 
evaluation effort thereby reducing some of the barriers 
described in the next section. 



Figure 3 

Locally Directed Evaluation 
Who ShouM Evaluate 



Context 



Who Should Be Involved In Conducting 
Evaluations? 

The answer to this third question will depend on the agen- 
cy conducting the evaluation However, those^schools and 
agencies which have conducted successful evaluations of 
their vocational programs and services tended to involve 
many groups of people. In fact, involvement and the will- 
ingness to participate seems to relate highly to useful 
evaluation activities. 

Representatives of the following groups typically can par- , 
ticipate in the IDE activities: 

1. Vocational Faculty 

2. Vocational Administrators 

3. Advisory Committee Members 

4. Student Services Staff 

5. Non-Vocational Administrators 

6. Vocational Students 

The representatives from these groups typically have cer- 
tain skills and abilities which can be quite useful to the* 
.evaluation efforts. These skills and abilities should be utiliz- 
ed. For example, vocational teSchers may be quite adept . 
at formulating objectives Tor the evaluation activity; ad- 
visory committee members may be excellent at develop- * 
ing letters or instruments mailed to groups outside of a 
school or agency; while vocational students can help co'iv 
duct many data gathering tasks. 
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Overcoming Possible Barriers, to Evaluation 

At times, and in some situations, there are real and im- 
agined barriers to conducting locally directed evaluation 
activities. Some of the more common barriers have. been 
identified through research and are: 

1. Staff Members' Time / 

2. Staff Members' Attitude *^ ' , 

3. Lack of Evaluation Expertise 

4. Failure to See How Evaluation Would Lead to 
Improvements 

Usually only one or two of these barners may apply in a 
local education agency. Yet, if they do exist, they need to 
be recognized and overcome. Again, based on research, 
here are some ways in which these barriers have been 
eliminated. 
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staff Wtombws'fimt 

Time is often a scarce resource. Occasionally, It is used 
as an excuse not to do sonnething. In sonne cases, while it 
is scarce we waste it, nevertheless. 

Some practical ways of dealing with this particular barrier 
be it real'or innagined, are: 

a/Get a verbal or written commitment from evaluation 
participants and those persons in authoritative positions 
within your school or agency that the evaluation activity 
is IMPORTANT. • * 

b. Spread the workload over many evaluation participants. 

a Develop short tasks which can be accomplished easily 
and quickly. 

d. Set realistic and firm completion deadlines for tasks 
and make certain everyone understands thenlr^ 

e. Use planning periods, office hours, lunch hours and 
other "non-scheduled'^ time to hold meetings and work 
on group tasks. 

f. Use substitutes, released tihie, reassigned time and in- 
service time to assist those working on the evaluation 
activity. 



Staff Mambers'^ Attitudes 

"Evaluation" is a term which has some emotional, and to 
some, threatening connotations to it. Some people fear 
evaluations while others are opposed to them for a 
number of reasons. A first reaction some people have to ^ 
evaluation is "what happens if the results are bad?" The 
possible answers one thinks of can interfere with the 
evaluation process either covertly or overtly. 

Figure 4 

Locally Dlr«cttd Evaluation 
Ovarcoming tha Baniars 
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Time 



Attitude 



Experience 



"What For*"? 




I /a e 1 1 1 n g 
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This negative attitude, if it exists, must be acknowledged 
and addressed. Some ways to deal with a negative at- ' 
titude are: 

a. Immediately indicate how ^valuation results will be 
used, letting everyone know the purpose of the evalua- 
tion effort. 
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b. Focus on the value of LDE for improvement of voca- 
tional progranns and services stressing a positive ap? 
proach. 

c. Reiterate the broad^sed involvement of many groups 
in the evaluation process to make it a fair and objective 
undertaking. 

d. Utilize someone from a different program or depart- 
ment within your school or agency who participated in 

* an LDg activity before to talk to the groqp about his or 
her experiences regarding the process. 

e. Utilize someone from a different school or agency who 
participated in an LDE activity before to talk to the 
group about his or her experiences regarding the pro- 
cess. 

f. Point to policies, goals, procedures or plans in your 
school or agency which mandate or suggest (depen- 
ding upon your situation) evaluation stressing the 
positive aspect of a /oca//y directed approach 
developed by the participants as opposed to one 
developed by some other external or internal group. 

Attitudes are difficult to change. The "threat" which 
sometimes is associated with evaluation can be reduced if 
discussed openly and honestly. The notion o'f a "hidden 
agenda" should be dispelled early in the evaluation pro- 
cess. The entire focus of Locally Directed Evaluation, as 
listed under the question, "Why Evaluate?" is for improve- 
' menL^ 



Lack of Evaluation Expertise 

Educators often have had little or no formal training in 
evaluation. They lack some technical skills associated with 
an evaluation endeavor. While some larger school 
districts,' community colleges,'and state agencies employ 
research or evaluation experts, most local education agen- 
cies do not have professional staff whose sole responsibili- 
ty is evaluation. 

These locally directed evaluation activities are designed 
for use by non-experts. They are organized to "guide" 
educators with little or no evaluation experience through a 
systematized evaluation exercise. Besides the materials, 
there are some actions which can be 'taken to overcome 
the real or perceived barrier of a lack of evaluation exper- 
tise. These actions are listed as follows: 

a. Contact the Consultant Section of the Illinois State 
Board of Education/Department of Adult, Vocational and 

' Technical Education for free assistance with the evalua- 
tion effort. 

b. Contact the Regional Vocational Administration (RVA) in 
your area of the state to discover which LEA's in your 
area have experience with LDE and could assi$t you 
with your evaluation efforts. 

c. Contact a nearby college or university to seek expert 
consultant assistance. 

d. Select an evaluation activjty which would appear 
familiar to many staff members (Example: Student 
Evaluation of Instruction). 

e. When assigning or suggesting evaluation tasks, try to 
assign experienced staff or those with some knowledge 
of evaluation with less experienced staff members. 

f. Analyze the evaluation activity you wish to pursue and 
determine if a portion of it coi^ be completed as a 
pilot project to give experience and confidence to 
evaluation participants. 
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FaHur* to Sm How Evtluatlon Would UM to 
NnprovMimis 

This particular barrier is sometimes associated with a 
negative attitude which doubts that change would take 
place regardless of evaluation findings. TTie failure or 
perceived failure of organizational change in the past is 
frequently the cause Of this attitude. This barrier is often 
manifested in the phrase, **wKat*s the use. nothing will 
change, anyway/' 

While resea/ch indicates that this barrier is not as fre- 

• quentty identified as some others, it is a difficult one to 
overcome. However, there are some actions which can- 
help to reduce this attitude. ' 

a. Secure adrriinistrative Qommitment to the evaluation 
undertaking t^efore it starts. Get a commitment that, if 
warranted, chgnge will take place. 

b. Involve key decision makers from all levels of the LEA's. 
organization in the evaluation process. Other evaluation 
participants will recognize the influence of the key deci- 
sion makers. 

c. Involve as many vocattonal faculty in the evaluation 
.process as possible. Since most LDE findings lead to 
changes in curriculum or programs, the faculty have 
the largest and most immediate impact oh these types 
of changes. 

d. Prepare publicity for internal and external publics an- 
nouncing the evaluation endeavor'before it occurs. 

^ Skeptical staff ipay realize the commitment to charige 
when public announcements are made about the 
evaluation effort even before it begins. 

e. Involve advisory committee members in the evaluation 
process. While these committees give "advice" and do 
not implement change, they are often a respected 
group whose involvement signals a serious effort to 
other staff members. 

^ f. Prepare an implementation plan for using evaluation 
results and present it to staff at the first meeting when 
' the evaluation effort is discussed. The plan should con- 
tain a process for using results, not the actual changes 
which will occur. The process will indicate a commit- 
ment to change which all staff should recognize. 

Barriers to the conduct of Locally Directed Evaluation 
(LDE) efforts exist in many LEAs. They need to he 

• acknowledged,''discussed and overcome. The uniqueness 
of each LEA will contribute to special barriers for an in- 

' dividual institution. The barriers described in this section 
have^been identified as fairly common ones based on 
research regarding LDE. They can be pvercome, without 
. greatiSfficulty. 



Procedure/Task 
Breakdown 

Task A. Establish a Tean/ for Planning and 
Coordinating the Evaluation. 

A team of individuals must be selected and appointed to 
assume responsibility for planning and coordinating the 
development of an evaluation activity. The team composi- 
tion will vary wilh the size and mission of the institution 
and its staff. However, generally speaking it is advan- 
tageous to involve a cross section of LEA personnel to 
foster the acceptance of the evaluation by everyone in the 
LEA. It may also be good to involve staff members that 
are respected by many and that ma/ facilitate the leader- 
ship of others. The following are suggested steps. 

1. Determine the desired representation. You might con- 
sider representatives of the following groups on the 
planning team: 

a. Administrators (principals, deans) 

b. Department chairpersons 

c. Instructors ^ 

d. Ancillary personnel (guidance, placement, research) 

e. Advisory committee members 

The number of individuals representing these 
categories will vary, but you should decide on a certain 
number basj^LiJi^n your local situation. For example, 
in a multi-attendancrtjeoler secondary distnct or a - 
community college, the planrtmg team composition 
might be: ^ 

1 Vocational director or dean \ 

2 Principals (secondary only) 
• 5 Department chairpersons 
.3 Instructors 

1 Director of institutional research 

1 Chairman of advisory committee 
A smaller institution might have a planning team with 
the following make-up: 

1 Local director or principal 

2 Department chairpersons 

^ 1 Advisory committee member 

2. Select individuals to match the desired team composi- 
tion. 

3. Invite the identified individuals to serve (chief school ad- 
ministrator, local director, dean or local leader may 
make this invitation). 

This can usually be accomplished through the use of a 
memo or letter. You can use Example Document' 1 as 
a guide. In some cases, invitation will be made jfer- 
sonally or in a faculty meeting. Flexibility is important ' 
here. 
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Task B/Schedule and Hold a Meeting of the Planning 
Team. 

1 Call a meeting of the planning team for the purpose of 
initiating work on developing the evaluation. 

2 Orient the planning team. It may be worth the time to 
summarize the questions raised in Document 6. 

3 Ask a high ranking superior (chjef school administrator, 
dean, direeior. etc ) to presents charge to the planning 
team and to indicate institutional commitment and ex- 
pectations related to the evaluation. 

4. Inform the team of its overall responsibility, 
a: To undertake an evaluation project that is designed 

to meet local needs for improvement, and 
b. To prepare a plan of action which depicts the pro- 
ject. 

5 Preseqt.an overview of the evaluation activity and 
discuss each mapr remaining task. Prior to team 

- discussions, ft is important that the leader review each 
task outlined in this guide. Example Document 2 is a 
transparency master prepared for use in presenting 
each task. The following outline may aid you in discuss- 
ing each task. 



Task B'1 Develop and Formalize a Purpose and Scope 
' Statement for the Evaluation, 

Subtask 1. Develop a purpose statement for the evaluation 
undertaking. 

Subtask 2. Determine the scope of the evaluation. 



Task d'2. Develop Key Questions, 

Subtask 1. These questions focus the evaluation to meet 
the needs and purpose identified in Task C. 

Subtask 2. These questions guide the entire effort. If 

something in the evaluation does not relate to 
one of these questions, then it is unwarranted. 



ERLC 



15 



15 



Subtask 3. Present several example key questions to the 
group on the overhead projector or on the 
chalk kx)ard. (Select several ex9mples=^fronr> Ex- 
ample Document 6.) " 

Task 8-3. Select Appropriate LDE Activities, 

Subtask 1 . Activities will be selected based upon their 

potential for answering key questions. 
. Subtask 2. Hand out and discuss list of activities 
(duplicate Example Document 3). 



Task 6-4. Obtain Resources for and Identify Constraints 
to Vour Evaluation Effort. 

Subtask 1 Estimate the resource needs for completing 
identified activities. 

Subtask 2. Identify existing resources, e.g., existing staff 
expertise, data processing services, eta 

Subtask 3. Obtain additional resources for the evaluation. 

Subtask 4 Identify constraining factors to t^e implementa- 
tion of the evaluation, e.g.. staff attitudes, union 
agreements, etc. 

Task 6-5 Prioritize Selected Activities in Light of ' 

Available Resources and Existing Constraints, 

Subtask 1 Delete activities for which resources do not ex- 
ist. 

Subtask 2. Delete activities that have constraints which 
cannot be overcome. 



Task 6-6. Identify Leaders and Establish Target Dates 
for Each LDE Activity. 

Task B-7: Project the Evaluation in the Form of a Written 
Plan. 



Task B'8. Initiate Implementation of the Evaluation Activity 
or Activities. 

6. 6uplicate, distribute and discuss the format for the out- * 
come of the evaluation. 

a. Duplicate and distribute Example Document 4 

b. Emphasize its simplicity. 

c. Emphasize and explain each item in the plan. 

7. Indid^te that the following tasks will be accomplished 
through a series of meetings. 



Task C. Develop and Formalize a Purpose and Scope 
Statement for the^^Evaluatlon. 

The first two items of the suggested plan format (Example 
Document 4) are a purpose statement and an indication of 
the scope for the evaluation. Much of the purpose will 
already be determined by you*and/or the chief school ad- 
ministrator. However, there is a need to have the planning 



team formalize the purpose and state it in writing. 
Simultaneous or integral to the development of the pur- 
pose statement is the determination of scope for the 
evaluation. You may actually define the scope within the 
purpose statement. 

1. Pr^ent your rough draft purposfe statement to ihe plan- 
ning team. This may be done on a duplicated sheet, or 
on an overhead projector and/or verbally. 

2. Ask the team for reactions and suggestions for improy- 
ing the purposp statement. Several ^xample^ purposes 
are^presented below. 

The purpose of the evaluation is: > 

to aid in identifying areas within our occupational offer- 
ings that need expansion. 

or 

to determine what aspects of our program need the 
greatest allocation of resources for improvement. 

or 

to improve occupational offerings. 

Your purpose statement might focus on on^or more of 
the following broad program planning or decision mak- 
ing questions. 

a. Where are you? (What are your resources, student 
and community needs, etc.'^ 

b. Where are you going? (What are your goals and ob- 
jectives?) 

c- How do you plan to get there?*(What strategies will 
you use to accomplish your objectives?) 

d. How are you doing in implementing your program? 
(Are you on target to achieve your goals and objec- 

• tives?) 

e. Have you arrived? (Have you achieved your goals ^ 
and objectives?) 

Exanryde Document 5 presents several example pur- 
pose/statements. 

3. Th/ statement of scope for the evaluation is easier to / 
d^elop than the purpose statement. The scope state-/ 
ment merely indicates to what part of the LEA the y 
evaluation will be applied, e.g. total Institution, occupa- 
tional proc/rams, a single occupational program, or a 
course. Initially, the scope statement will focus on total 
occupational program offerings. Later, you may wish to 
develop or have each department or unit develop its 
own evaluation plan. An example scope statement is 
presented below. 

The scope of the evaluation is: 

to focus the evaluation effort on the total occupational 
program of our LEA. 



Task D. Develop Key Questions 

Key questions are broad questions which focus the 
evaluation. They take the general or overall purpose state- 
ment and begin to make it more specific. Activities and ef- 
forts should all be selected to a'nswer key questions. This 
places great emphasis on care and attention to the 
development of adequate key questions. 
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Program impfoverrient needs may pe the most \^luable 
source for identifying key questions. General cpncems or 
feelings- of staff may reveal some of these, but other 
• source^ should also t>e considered. State evaluation 
reports and regional accreditation recomnf>endations can 
provide indications of program components whictj may 
need further analysis or evaluation. Six. areas or com- 
ponents of a vocational program have been identifjed as 
being vital to a successful program. These compdhents 
a/e: 

Program Management 

Personnel < ' 

Ranning and Evaluation 
l^esources UtiTized 
Student Sitrvices 
Program Content 

These components can be considered with the five pro- 
gram planning or decision making* questions to aid in the 
formulation of key questions Example Document 6 pro- 
vides example key questions classified according to the' 
five planning questions mentioned in Task C, step 2. 

The following is an outline for the local leader to follow in 
helping the team formulate key questions. These steps 
may be altered to meet your local needs. 

1. Build and present a Rationale for having key questions. 

a. To focus the evaluation 

b. To aid in selecting evaluation activities 

c. To aid in developing instruments 

d To aid the utilization of evaluation -results 

2 - Present possible sources of key questions. 

a ^Previous evaluation reports (accreditation, state 

visits, etc.) 
b. Concerns of staff - 
c" Planning decisions and needs 

d. Areas of concern 

e. Five planning or decision making questions. 

3 Show transparency of fjy^ planning questions. (Exam- 
ple Document 7) 

4 Duplicate example key questions and distribute to 
planning team. (Example Document 6) 

5. Duplicate or obtain multiple copies of previous on-site 
evaluation reports (accreditation, state, etc.) or other ' 
informational documents which might help tn identify- 

• ing key questions. 
^ 6. It may be advantageous if you, the local leader, 
highlight some of your priority concerns from any of 
the five points under item 2 above. This n\ay give the 
planning team a starting point. 

7 Depending on the size of the planning team, -as a 
group or as divided sub-groups, write some potential 
key questions, (k'eeping in mind the overall purpose 
and scope statements), tou may wish to do this out- 
side of your current rq eeting and ask the t eam 
members to bring sorne queSiions to tne?lext 
meeting. ^ 

8 Later in the meeting or at the next meeting of the 
team, review several questiohs and begin discussing 
each. 



9. You will probably have more questions than you can 
feasibly answer within the resource limits for evalua- 
tion. Initially, it Js tetter Jo have a long list of questions 
that you can prioritize }ater. Don't be afraid to adapt 
questions from the examples provided, but don t/eiy 
on them entirely. Remember, the evaluation that you 
are designing should meet your specific needs. 

10. Duplicate the list of key questions developed by the 
team. Then, if the list is long, have the teana begin to 
obtain team reaction atxxjt priority questions. 

11. The product of thislask should be a list of questions 
that will guide further development of the evaluation 
effort. 

12. You may wish to have a^ consultant, j^dvisory commit- 
tee or supen/isors react to your list of key questions 
at this time. 



Task E Select Appropriate LDE Activities. 

The LDE activities 1 through 12 make up the remainder of 
this set and have been designed to aid m ^nswenng the 
key questions developed in Task D. The following sugges- 
tions are offered to aid in activity selection. 

1 . As a leader, become familiar with Bach LDE activity. 
Additionally, you may wish to review each Local 
Leader Guide to better understand the purpose and 
procedures. 

2. Discuss the activities, etjiphasizing'the potential pur- 
poses and major tasks* of each. 

3. Consider each activity in light of key questions. You, 
as the leader, may identify several activities prior to 
group discussion and then have the planning team 
'react to your choices ahd suggest other or additional 
activities. Team members may want clarification or 
more information about individual activities. You 
should hgve your set of Local Leader Guides available 
for consultation. 

4. You may end up with more activities than you can 
PQSSibJyj^omplete. Therefore, you should, through 
discussion, determine which activities are viewed as 
most important. Then place the list of selected ac- 
tivities in priority order. 

Task R Obtain Resources for and Identify Constraints 
to Your Evaluation Effort. , ^ 

,/ 

The development and implementation of an evaluation will 
consume resources sucli^s personnel time and money. 
However, with a commitment from superiors to conduct 
an evaluation must also come a commitment to provide 
resources. This fact reinforces the involvement of the 
superiors in Tasks A and B..You may know what 
resources are available tor the ^valuation prior to the con- 
sideration of this task. If this is the case, proceed directly 
to Task G. If not, consider the following steps. 

This task involves the listing of key questions (from Task 
D), the listing of selected activitiesTftaf match key ques- 
tions, and the estimation of resource requirements for 
each activity. Then a request for riecessary resources is 
made to superiors. The last aspect of the task involves the 
identification of constraining factdrs to implementation jpf 
the evaluation endeavor. 
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1, Combine Ihe list of key qu^tions the team developed 
in Task D with the activities you identified m Task E. 
Present this dual listing in prionty order using the 
following format (Example Document 8). 

Key Question 1. Activity # 2, 3 

Key Question 2. Activity # -5 

Key Question 3. Activity # 2 

2, Estimate Jhe time and'dollar resources necessary fjor 
each activity. This can be done by reviewing each 
Local Leader Guide and considering each task within 
the activities, placing their estimated amounts in the 
third and fourth columns 6f the form shown in Example 

, Document 8. 

3 , Present the purpose and scope staterhents and your 
cofppleted document to 4Ke person who has financial 
authority to inform him/her of the planning team's ef- 
forts. At this time, obtain a reaction from the superior 
about the availability of resources for the evaluation 
(This may appear to be premature but it is important to 
know, what is available prior to nnaking final plans for 
the evaluation effort.) 

4 Identify any internal or external resources that may be 
available for the evaluation. These might include* 

a Existing staff expertise 

b Data processing services and facilities 

c. Related research efforts * 

d Other 

5 Identify' constraints to the irriplementation of the evalua- 
tion. If any exist. Items such as the following might be 
included: 

a. Bad experience ^ith evaluation in the past 

b. Staff attitude toward evaluation 

c. Union or teacher organization policy 

d. Repcioritization'of work loads 

e. Other 



task G. Prioritize Selected Activities in Light of 

Available Resources and Existing Constraints. 

Ideally, you will identify and obtain resources that are ade- 
quate to focus on all identified ke>iquestions and to con- 
duct each of your selected, activities. If this is the case, 
proceed to Task H. Howfever, if available resources are 
less than adequate, it will be necessary for you to select 
the high priority activities. Also, it is important to consider 
and discuss constraints. If constraints can be overcome, 
the actions should beJaken to do so. If they cannot be 
overcome, then certsnn activities may need to be deleted.' 

1. From the summary form (Example Document 8), iden- 
• tify those activities that can be completed with 

resources available. 

2. Identify activities that hgye insurmountable constraining 
fa&tofs and delete them from Example Document 8. 



Task>1. Identify Leaders and Establish Target Dates 
for Each LDE Activity. 

1. From the planning group preferably, identify an in- 
dividual to be responsible for leading each of the 
selected activities. 
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, 2. Expertise in the Jopic is not a necessary criterion for 
selection since Local Leader Guides assume no 
previous experience in conducting activities. 

3. Once personnel are identified, either by choosing 
volunteers or by selecting others, they should be given 
the respective Local Leader Gukie. 

4. After each leader has had the chance to review the 
Local Leader Guide, an estimate of a reasonable star- 
ting and completing date should be established for the 
activity. 

5. Each leader should proceed with activities and 
establish teanns. 



Task I. Project the Evaluatioa in the Fonn of a Written 
Plan. 

The plan is a concise document that can be used to com- 
municate the vanous aspects of the evaluation to instruc; 
tional and supportive personnel. The form presented in Ex- 
arriple Document 4 is suggested. 

1. Race the purpose statement that was developed it] , 
Task C on the plan document (Example Document 4). 

2. Put the scope statement, also from Task C, in the ap- 
propriate place on the plan document. 

3. List prioritized key questions developed in Task D and 
prioritized in Task G. 

4. List or summarize an estimate of available resources, 
from Task F, on the plan document. 

5. Ust or summarize p,onstraints, from Task F, on the plan 
document. 

6. List selected activity titles, identified leaders for each, 
and estimated beginning and completing dates. 

Task J. Initiate Implementation of the Evaluation 
Activity or Actfvtties. 

The plan development is only the beginning of the evalua- 
tion effort. The evaluation must be initiated, monitored and 
evaluated. The following are suggestions for initiation*. The 
list is far from complete. 

1. Ch^k your written procedural plans jpr initiation to en- 
sure that you have considered the following essentia] 
features for a successful evaluation^ 

a. Commitment 

b. Improvement purpose ' 

c. Involvement - 
ds^road base 

e. Systematic recording 

f. Judgment \ 

g. Feedback 

J>rComprehensive focus ^ 
i. Using results ^ 

2. All staff in the LEA s^ld be informed that an evalua- 
; ton is^underway. ^ v 

3f. leaders of each activity should establish teams and for- 
mulate a plan for completing their activity. 

4. The l8cal leader should monitor the progress.of each 
activity. 
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Using Results 



The Locally Directed Evaluation (LDE) materials contained 
in this book lead to a series of evaluation activities. Their 
influences for constructive change and improvement rests, 
to great extent, on the use of the results. Evaluation tasks,' 
materials and example documents are provided for a sue-' 
cessful evaluation endeavor. Also, each separate evalua- 
tion booklet culminates with tasks and materials which 
deal with using the results of the specific evaluations. 
However to insure that the LDE activity is not just an 
"academic execise" with hp tangible results, additional in- 
formation on using evaluation results is provided in this 
section of the User's Guide. 



Distributing Evaluation Findings 

• 

This guide suggests that a broad base of people be involv- 
ed in conducting evaluation activities (Figure 3) Many of 
these same people along with other groups should receive 
copies or summaries of evaluation findings. Each of these 
groups can contribute in some way to the implementation 
of suggestions-for improvement. A survey of over 200 
LEAs indicated that the groups listed in Figure 6 con- 
sistently received information about evaluation findings 



~ Vocatlofial Faculty 

This group should be directly involved with the evaluation 
process from beginning to end. Those members not in- 
^volved in a specific evaluation activity need to be informed 
of evaluation findings regarding all parts of the agency s 
total vocational program. This information will assist them 
in planning and improving programs and advising students. 



Coun9elor8 

Counselors, when informed of evaluation findings, should 
be in a better position to advise vocational students. They 
also can contribute to program improvement activities 
vyhen they are aware of the status of current vocational 
prdgrams. 



Admini8tratoi*s 

This group\s a key one in decision making concerning 
vocational programs. In a study of these LDE activities, 
the higliest rate of perceived success with using evalua- 
tion results was "for informing administrators/ Therefore, 
administrators need to know aboiit evaluation findings in ' 
order to help support and develop improvements in the 
total vocational program. 



Advlsoiy Committees 

Advisory committees are influential and knowledgeable 
groups which can help bring about change. They can 
make important suggestioris and recommendations for im- 
provement when they receive results of evaluations and 
are involved in program improvement activities. 

f\guc 6 

Locally Directed Evaluation 
Groups Receiving Evaluation Results 




Board of Control 

The LEA board of control should be informed of evaluation 
activities and the results of these activities. Policy deci- 
sions, expenditure approvals and planning considerations 
made by boards, impact the total vocational education 
program Board members need good information, which is 
provided by evaluation findings, on which to make sound 
decisions. 



Public 

Vocational education has a story which ought to be told to 
the public A summary of vocational evaluation findings 
should be transmitted through various media to the LEA'S 
public Ultimately, all educational decisions are accepted 
or rejected by the public. They need to be informed of 
vocational education as much as any other group. 

Tbe various groups whicfjl should receive information from 
these Locally Directed l?valuation (LDE) activities may or 
may not be directly involved with using the results of the 
findings Those who are involved with making recommen- 
dations or implementing change need information to ac- 
complish these tasks successfully. Others, need informa- 
tion to be informed and kept current about vocational 
. education programs and services. 



Using Evaluation Rndings 

Each of these 12 evaluation activities can lead to specific 
changes and improvements in vocational education pro- 
grams and services. Individual activity findings will suggest 
various recommendation for change. Each activity also 
contains tasks and a format for initiating change. A listing 
of the ways in which local evaluation results have been 
utilized by LEAs appears in Figure 7. These ways are , 
listed in rank order of use. 



Changing the Cuniculum 

Vocational education learning activities are at the core of 
all vocational programs. These learning experiences, 
organized into a curriculum or program, are often im- 
pacted by the results of LDE. activities. Changes in existing 
courses, adding new courses and deleting others are fre- 
quently recommended based on evaluation findings. When 
these changes result from empirical evidence, they lead to 
improvement. 

Informing Administrators 

Most changes, and certainly major ones, in vocational 
education programs and services need to be approved or 
at least supported by administrators. They need to have 
information to support decisions. Even if evaluation find- 
ings do not suggest any major changes, administrators 
should be appraised of the current status of vocational 
education in an LEA. 



Staff Development 

Evaluation findings can lead to valuable staff development 
experiences. The vocational staff and other key individuals 
can discuss the findings, make suggestions for improve- 
ment and actually work on improvement activities as staff 
development experiences. Therefore, LDE findings should 
be integrated into a local education agency's staff 
development pr£)gram. 



Equipment Requests 

Supporting documentation for equipment purchases may 
emanate directly from some LDE activities and indirectly 
from others. Specific findings regarding the need for 
equipment changes, deletions or additions may result from 
specific evaluation activities. If equipment needs to be ad- 
ded or up-dated, the justification will be determined by the 
evaluation findings. ' 



Recruiting Studeojs . 

Successful vocational programs and services are needed 
to attract students. When evaluation bindings demonstrate 
successful efforts, these findings should be communicated 
to the public, particularly to potential student^. Local 
education agencies such as ar6a vocational centers and 
community colleges can use evaluation findings to recruit 
students who are not in attendance at the institution 
thrnuah external communications. Secondary schools, unit 
rn^ s and state agencies can us6 internal communica- 



tion vehicles to recnjit students to vocational programs 
v/ho may be involved In the educational agency but not 
enrolled in a specific vocational program. A method of 
communicating vocational evaluation findings for the pur- 
pose of recnjiting students should be an integral part of 
using the evaluation findings. 
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Locally Dir»ct»d Evaluation 
Using Evaluation Rndlngt 



Using 
Evaluation ^ 
Findings 




Changing Curricula 




Informing Adrninistrators 




Staff Development 




Supporting Equipment Requests 




Recruiting Students 




Discontinuing- p/ograms ' 









Discontinuing Ptogrims 

At times evaluation findings point toward the discontin- 
uance of a vocafional program or sen/ice. Such a decision 
is frequently a drastic one which could affect staff, 
students, resources and the^ agency's mission or purpose. 
The LDE activities provide a rationale for this decision 
when it is warranted. 

In almost every case, the conduct of a Locally Directed 
Evaluation activity will provide findings which can lead to 
change and improvement. It is important to utilize the fin- 
' dings so constructive change does take place. This sec; 
tion of the User's Guide illustrates ways in which evalua- 
tion results have been utilized by LEAs. Many of these * 
agencies developed a comprehensive action plan for utiliz- 
ing results. The following sectionjncludes methods by 
which LEAs insure that the results of evaluations are used 
for improvement purposes. 

Insuring the Use of Results 

.Completing an LDE activity by identifying conclusions, 
recommendations and suggested improvement activities 
will not guarantee improved vocational progranjs and ser- 
'vices. Action must be initiated to implement results. Con- 
structive change needs to be undertaken as soon as the 
evaluation effort is finished. The change process can be 
facilitated by using the techniques identified in Figure 8. 
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HordM««tings 



Holding meetings with individuals who can help initiate 
change Is an important component of the change process. 
Staff meetings, advisory committee meetings. oneK)none 
meetings with key individuals and staff development ses- 
sions focused on the results of the evaluation effort will 
lead to activities aimed at improving vocational education 
programs and services. Periodic status or up<late 
meetings will help keep the improvement process active 
and functioning* 

Produce Written Procedures 

Written procedures are related to the meetings which 
ought to be held as part of the change process. The pro- 
cedures should be a blue-print for change. They should 
specify the following: 

—improvement activities to be undertaken 
— who is to be involved 
—when the activities should be accbmplished 
. ,— the evaluation measure required to determine the suc- 
cess of the activity 

Specific written procedures for improvement can be 
t developed for each evaluation activity and the forms and 
tasks for these procedures are included in^ach booklet. 



Build into Planning 

Locally Directed Evaluation results can be built into the 
LEAs planning system. Building the results into the One 
and Five Year Local Improvement Plan for Vocational 
Education and any other internal planning system will help 
insure that evaluation results are utilized. The planning 
system should drive the improvement efforts so that they 
become a part of the agency's operating routine. Change 
and improvement will then become an acceptable, com- 
mon process which is a logical extension of evaluation. 

Assign to Appropriate Persons 

This component, of the process for insuring that results of 
evaluations are used, is a vital one. Improvement requires 
action on the part of some person or persons. The in- 
dividuals involved in the evaluation activity should identify 
the key people needed to implement specific recommen- ' 
dations or suggestions. These persons chould be given an 
assignment and the resources needed to complete their 
tasks. The individuals most directly related to the recom- 
mendation or suggestion should be involved with its im- 
plementation. . 
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Locally DIrtct»d Evaluation 

Insuring tht Ut«of Evaluation Rttults 




Evaluation Results Model 

Figure 9 illustrates a Model for using the results of LDE 
Endeavors. This process can be applied to any of the 12 
activities in this evaluation systerii. Figure 10 is a afample 
Action Plan format for use with the Evaluation Results 
Model. It relates directly to activity number 5 in the Model. 

Each of the 12 separate evaluation activities in the Locally 
Directed Evaluation system includes a culminating Task 
Breakdown which deals with using the results of that 
specific activity. Normally, this final Task Breakdown cor- 
responds with activity number 2 in the Evaluation Results 
Model (Figure 9). The Model should be followed for each 
LDE activity or combination of evaluation activities under- 
taken by an LEA. Such action should guarantee that 
evaluation will make a substantial' contribution to program 
improvement. 



Figure 9 

Locally Directed Evaluation 
Evaluation Results Model 
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Figure 10 

Locally Directed Evialuatlon 
Act bn Plan Format 
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Example DocumiBnts 




No. 



Page Number 



1. Memo— Invitation to Serve on Planning Team \ ... * - - . 2A 

2. Transparency Master of Major Tasks . < . , 25 

3. List or 12 Activities * . * ^ ^ 26 

4. Suggested LDE Plan Format ^ ^ 27 

5. List of Example System Purpose Statements 28 

6. ..Example Key Questions Categorized by Planning Questions ' ^29 

7. Transparency Master of Five Planning Questions 30 

8 Listing of Prioritized Key Questions, LDE Activities. Estimates of Personnel Time, and Estimates of Needed 

Dollar Resources 01 
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Example Document 1 * « 



t5 

MEMO—INVITATION TD SERVE ON PLANNING TEAM 



MEMO 



TO: Prospective Evaluation System Planning Team Member 

FROM: Chief ^School Admi'filstrator 

RE: Agency Evaluation System Development 



As you are aware, there is an ever increasing need for educational 
institutions to formalize their evaluation procedures into Systems that 
annually provide adequate data for^ecision making. During past years, 
we have been quite involvisd in various evaluative efforts such as accred- 
itation, self studies, follow-up Studies, staff evaluation, and student 
evaluation of instruction. But, as of yet, we have-not united activities 
into one evaluative effort'that provides us with data for agency-wide and' 
departmental program planning. 

To help overcome this shortcoming, we need to establish a formal eva>ua- 
tipn system. We need a system that provides the data needed to enhance 
decision making by our instructional and administrative staffs as well as 
to provide results to the community concerning the'utilizatioa of publi.c 
funds. • . 

>I would like to invite you to be a member of the Evaluation System 
Planning Team for our agency. Your assistance in this matter would be 
greatly appreciated. "Wie firs-t meeting of this group is scheduled for 
2:30 p.m, in the Board Room on the 22nd of this, month. Ple^e respond to 
my office prior to the 15th concerning your availability to at/^end this 
meeting. 

Thank you for your consideration to this most important matter concerning 
the future of our educational irtstitution, ' 
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Example Document 2 




TRANSPARENCY MASTER OF MAJQR TASKS 



Task A. Establish Team 
Task B. Schedule Meeting 
Task C. Develop Purpose and 
Scope 

Task D. Develop Key Questions 
Task E. Select LDE Activities 
Task F. Obtain Resources arid 

Identify Constraints 
Task G. Prioritize Activities . 
Task H. Identify Leaders and 

Establish Dates 
Task I. Profept Written Plan 
Task J. Initiate Implemehtation 



ERIC 
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Example Document 3 s 

Ust of 12 Acttvtties 

Activity 1. Studenf Evaluation of Instruction 

Activity 2. Student/Employfer Follow-up 

Activity 3. Measuring Student Competencies 

. Activity 4. Assessing Basic Skills 

Activity j5. Ass)^ssing Student Services ^ 

Activity 6. ^Assessing Special News Services 
and Identification Practices 

Activity 7. Assessing Access and Equity 

Activity "8. Evaluatirig Facilities 

Activity 9. Evaluating Instructional Materials 

Activity lb. CostOutcome Analysis. 

Activity 1 1, Analyzing Community Resources 

Activity 12. Assessing Community Needs • 

i 



; 



Example Documents 

Suggested Cde Pian Format 

Purpose: 



Scope: 




ey Questions: 



Resources: 



Constraints: 



^^etiyiM Leaders ' , Target Dates 

> Begin End 
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Example Document 5 • 
' List of Example System Purpose Statements . 

The purpose of the evaluation system is: 

1. To determine what aspects of our total occupational program need improvement. 

2. To analyzQ the program to determine needed curriculum content revisions. 

3. To determine how well orientation courses prepare students for training level courses. 

4. To assess occupational program graduates concerning their impressions of the program that they com- 
pleted. 

5. To determine staff in-service needed for program improvement.. 

6. To gather sufficient data to enhance occupational program planning decision making. 

7. To assess occupational supply and demand information m light of needed program additions and deletionis. 

8. To provide sufficient data to aid the instructional and administrative staff as well as the Board of Education 
in their decision making functions. 

9. T(^ gather sufficient data to inform the public of outcomes of educational expenditures. 

10. To aid in identifying areas which need expansion wrthin our occupational offerings. 

1 1. To determine what aspects of our program need the greatest allocation of resources for improvement. 

12. To improve occupational offerings. 
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Example Document 6 

Example Key Questions Categorized by 
Planning Questions 

Potential 

LDE 

Activity 

Where are you? 

(2) (12) a. What are the personnel needs for various 
occupations within the community? 
(5) b. What are the career interests of' students 
within the educational agency? 

c. What competencies are held by staff 
• members? 

d. What resources exist in the community for 
offering career programs? 

e; What in -school facilities exist forjDrogram 
operations? < 

Is the present curriculum compatible with 
student interest? 

Are students interested in present courses? 
Can instructors better structure their daily 
plans to focus on individual student career 
Interests? 

(8) i How effectively are in-school facilities and 

equipment being utilized? 
^ (8) j Is there over or under utilization of existing 
space? 

(8) k. Are safety standards being adhered to? 
(8) I. Is the existing learning environment con- 
ducive to learning? 

(8) m Is there a need for planning facility expan- ' 

sion? 

(10) n. What are the cost and benefit measures for 
^ prpgrams? 

(9) 0 Are the current job competencies required , 

for entry into. various occupations? 

2, Where^re^you^ng^-^ 

(3)* a How yell are students achieving program 
and course objectives? 
f1-2h-(3)— b^Do-established^pjjDgram aj^ course objec- 
tives correspond to required job compeTeh* 
cies? 

How will you get there? 

How realistic are our educational improve- 
ment plans? 

Do educational plans reflect the concerns of 
staff? 

Do programs provide adequate preparation 
for employment? 

Are courses sequentially structured ihto pro- 
grams? 

How might the format of instructional objec- 
tives be improved? 

How might the measurement instruments 
and procedures for assessing student attain- 
ment of objectives be Expanded? 





3. 




a. 


(2) 


b 


(2) 


c. 


. (2) 


d. 


(3) 


e. 


(3) 


f. 



Potential 

— tDE 

Activity 

4. How are you doing In getting there? 

(1) a, How do students evaluate the instruction* 
' they receive? 

b. What are staff needs for in-service training? 

c. How can \he administrative structure be im- 
proved? 

d. V/ha\ communication problenns exist in Ihe 
administrative organization? 

(1) e. How well do instructors prepare. program, 

course and lesson plans? 
(1) (7) f. Do instructors allow for individuaf student 

differences? 
(1) g. How well do instructors motivate their 

students? 

(1) (2) * h, Do instructors possess a command of the 
subject they teach and is it in tune with the 
current world of work? 
(1) 1. Do instructors utilize effective teaching 
techniques? 

(5) J. Is our counseling service meeting the needs 
of our students? , 

(5) k. Are our student records complete and ac- 
curate? 

(5) L Is our student testing program broad enough 
in scope? 

(5) m.ts pur LEA's placement program functioning 
effectively? 

(5) n. Are our LEA's youth organizations of value 
and do they have a positive impact on our 
students? 

(5) 0. Is our LEA'S program of information'about . 
careers closely articulated and coordinated 
with feeder agencies and agencies that 
students will next attend? 
(5) p. Has 4he career information program had a 
demonstrated and positive impact upon stu- 

dent career and educational decisions? 

(6)~7T7) qTT^helnterests^f-speeialH^eds-studeQtS- 

being met by occupational programs? 
5. Have you made It? 
(2) a. What do former students think about the 
training they received while attending the 
LEA? 

(2) , b. Are students employed in the area for whteh 
they were trained? 

(2) c. What are employers* impressions of training 

received by employees trained -by otfr in- 
' stitution? 

(3) d. To what degree do students achieve the 

performance objectives established for 
them? 

(2) e. How do employers rate students coming 
^ from a training program as compared to 
those who did not go through such a pro* 
granri? ' ' 
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TRANSPARENCY MASTER OF FIVE PLANNING QUESTIONS 



1 .i//here are you? 



2. Where are you going? 

3. How do you plan to get 
there? 

4. How are you doing in 
implementing your 
program? 

5. Have you arrived? 
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Key Questions 



highest 
priority 1. 



Activities 



Estimate of 
Personnel 
Time • 



Estimate 
of Needed 
Dollars 



2. 
3. 
4. 
5. 



32 



6 



IIIInoi$ 

State Board of 
Education 

100 North First Street 
Springfield, Illinois 62777 



Edward Copeland 
Chairman 



Donald Gill 

State Superintendent of Education 



Ah Equal Opportunity Employer 



